Extreme image transformations affect humans and machines differently.

Some recent artificial neural networks (ANNs) claim to model aspects of primate neural and human performance data. Their success in object recognition is, however, dependent on exploiting low-level features for solving visual tasks in a way that humans do not. As a result, out-of-distribution or adversarial input is often challenging for ANNs. Humans instead learn abstract patterns and are mostly unaffected by many extreme image distortions. We introduce a set of novel image transforms inspired by neurophysiological findings and evaluate humans and ANNs on an object recognition task. We show that machines perform better than humans for certain transforms and struggle to perform at par with humans on others that are easy for humans. We quantify the differences in accuracy for humans and machines and find a ranking of difficulty for our transforms for human data. We also suggest how certain characteristics of human visual processing can be adapted to improve the performance of ANNs for our difficult-for-machines transforms.

Mitigation of Effects of Occlusion on Object Recognition with Deep Neural Networks through Low-Level Image Completion.

Heavily occluded objects are more difficult for classification algorithms to identify correctly than unoccluded objects. This effect is rare and thus hard to measure with datasets like ImageNet and PASCAL VOC, however, owing to biases in human-generated image pose selection. We introduce a dataset that emphasizes occlusion and additions to a standard convolutional neural network aimed at increasing invariance to occlusion. An unmodified convolutional neural network trained and tested on the new dataset rapidly degrades to chance-level accuracy as occlusion increases. Training with occluded data slows this decline but still yields poor performance with high occlusion. Integrating novel preprocessing stages to segment the input and inpaint occlusions is an effective mitigation. A convolutional network so modified is nearly as effective with more than 81% of pixels occluded as it is with no occlusion. Such a network is also more accurate on unoccluded images than an otherwise identical network that has been trained with only unoccluded images. These results depend on successful segmentation. The occlusions in our dataset are deliberately easy to segment from the figure and background. Achieving similar results on a more challenging dataset would require finding a method to split figure, background, and occluding pixels in the input.

Tuning Properties of MT and MSTd and Divisive Interactions for Eye-Movement Compensation.

The primate brain intelligently processes visual information from the world as the eyes move constantly. The brain must take into account visual motion induced by eye movements, so that visual information about the outside world can be recovered. Certain neurons in the dorsal part of monkey medial superior temporal area (MSTd) play an important role in integrating information about eye movements and visual motion. When a monkey tracks a moving target with its eyes, these neurons respond to visual motion as well as to smooth pursuit eye movements. Furthermore, the responses of some MSTd neurons to the motion of objects in the world are very similar during pursuit and during fixation, even though the visual information on the retina is altered by the pursuit eye movement. We call these neurons compensatory pursuit neurons. In this study we develop a computational model of MSTd compensatory pursuit neurons based on physiological data from single unit studies. Our model MSTd neurons can simulate the velocity tuning of monkey MSTd neurons. The model MSTd neurons also show the pursuit compensation property. We find that pursuit compensation can be achieved by divisive interaction between signals coding eye movements and signals coding visual motion. The model generates two implications that can be tested in future experiments: (1) compensatory pursuit neurons in MSTd should have the same direction preference for pursuit and retinal visual motion; (2) there should be non-compensatory pursuit neurons that show opposite preferred directions of pursuit and retinal visual motion.

Computational modeling of depth ordering in occlusion through accretion or deletion of texture.

Understanding the depth ordering of surfaces in the natural world is one of the most fundamental operations of the primate visual system. Surfaces that undergo accretion or deletion (AD) of texture are always perceived to behind an adjacent surface. An updated ForMotionOcclusion model (Barnes & Mingolla, 2013) includes two streams for computing motion signals and boundary signals. The two streams generate depth percepts such that AD signals together with boundary signals generate a farther depth on the occluded side of the boundary. The model fits the classical data (Kaplan, 1969) as well as the observation that moving surfaces tend to appear closer in depth (Royden, Baker, & Allman, 1988), for both binary and grayscale stimuli. The recent "Moonwalk illusion" described by Kromrey, Bart, and Hegdé (2011) upends the classical view that the surface undergoing AD always becomes the background. Here the surface that undergoes AD appears to be in front of the surrounding surface-a result of the random flickering noise in the surround. As an additional challenge, we developed an AD display with dynamic depth ordering. A new texture version of the Michotte rabbit hole phenomenon (Michotte, Thinès, & Crabbé, 1964/1991) generates depth that changes in part of the display area. Because the ForMotionOcclusion model separates the computation of boundaries from the computation of AD signals, it is able to explain the counterintuitive Moonwalk stimulus. We show simulations that explain the workings of the model and how the model explains the Moonwalk and textured Michotte phenomena.

Neural dynamics of feedforward and feedback processing in figure-ground segregation.

Determining whether a region belongs to the interior or exterior of a shape (figure-ground segregation) is a core competency of the primate brain, yet the underlying mechanisms are not well understood. Many models assume that figure-ground segregation occurs by assembling progressively more complex representations through feedforward connections, with feedback playing only a modulatory role. We present a dynamical model of figure-ground segregation in the primate ventral stream wherein feedback plays a crucial role in disambiguating a figure's interior and exterior. We introduce a processing strategy whereby jitter in RF center locations and variation in RF sizes is exploited to enhance and suppress neural activity inside and outside of figures, respectively. Feedforward projections emanate from units that model cells in V4 known to respond to the curvature of boundary contours (curved contour cells), and feedback projections from units predicted to exist in IT that strategically group neurons with different RF sizes and RF center locations (teardrop cells). Neurons (convex cells) that preferentially respond when centered on a figure dynamically balance feedforward (bottom-up) information and feedback from higher visual areas. The activation is enhanced when an interior portion of a figure is in the RF via feedback from units that detect closure in the boundary contours of a figure. Our model produces maximal activity along the medial axis of well-known figures with and without concavities, and inside algorithmically generated shapes. Our results suggest that the dynamic balancing of feedforward signals with the specific feedback mechanisms proposed by the model is crucial for figure-ground segregation.

Modeling a space-variant cortical representation for apparent motion.

Receptive field sizes of neurons in early primate visual areas increase with eccentricity, as does temporal processing speed. The fovea is evidently specialized for slow, fine movements while the periphery is suited for fast, coarse movements. In either the fovea or periphery discrete flashes can produce motion percepts. Grossberg and Rudd (1989) used traveling Gaussian activity profiles to model long-range apparent motion percepts. We propose a neural model constrained by physiological data to explain how signals from retinal ganglion cells to V1 affect the perception of motion as a function of eccentricity. Our model incorporates cortical magnification, receptive field overlap and scatter, and spatial and temporal response characteristics of retinal ganglion cells for cortical processing of motion. Consistent with the finding of Baker and Braddick (1985), in our model the maximum flash distance that is perceived as an apparent motion (Dmax) increases linearly as a function of eccentricity. Baker and Braddick (1985) made qualitative predictions about the functional significance of both stimulus and visual system parameters that constrain motion perception, such as an increase in the range of detectable motions as a function of eccentricity and the likely role of higher visual processes in determining Dmax. We generate corresponding quantitative predictions for those functional dependencies for individual aspects of motion processing. Simulation results indicate that the early visual pathway can explain the qualitative linear increase of Dmax data without reliance on extrastriate areas, but that those higher visual areas may serve as a modulatory influence on the exact Dmax increase.

A computational study of brightness-related responses in visual cortex.

Rossi, Rittenhouse, and Paradiso (1996) reported a cut-off at 4 Hz in the modulation amplitude of neural responses to large (up to 14°) simultaneous contrast stimuli in the striate cortex of cats, as the temporal frequency of the luminance of flanking patches increased, while the luminance of a central patch covering the neurons' classical receptive fields (CRFs) was held constant. This indicates that the modulation may involve slow processing of information in visual cortex. We develop models with a small set of parameters to explain these brightness-related responses in visual cortex. A model with any of the following mechanisms can fit the data: (a) slow local inhibition (Slow Inhibition Model); (b) slow excitation of the nodes in the second of two layers, which feed back to the inhibitory nodes in the first layer (Slow Excitation Model); and (c) conduction delays along lateral connections (Delay Model). However, the Slow Inhibition Model predicts that neurons in extrastriate cortex show similar response modulations as neurons in V1, while the Slow Excitation Model predicts that, unlike the modulations of V1 neurons shown in the experimental data, neurons in extrastriate cortex show slow modulations of responses to both the direct luminance change and the simultaneous contrast stimuli. The Delay Model predicts that the cut-off frequency of the response modulations depends on the distance from the flanker to the CRFs of the neurons. Further physiological experiments could clarify which mechanism plays an important role in the brightness-related responses in visual cortex.

A neural model of visual figure-ground segregation from kinetic occlusion.

Freezing is an effective defense strategy for some prey, because their predators rely on visual motion to distinguish objects from their surroundings. An object moving over a background progressively covers (deletes) and uncovers (accretes) background texture while simultaneously producing discontinuities in the optic flow field. These events unambiguously specify kinetic occlusion and can produce a crisp edge, depth perception, and figure-ground segmentation between identically textured surfaces--percepts which all disappear without motion. Given two abutting regions of uniform random texture with different motion velocities, one region appears to be situated farther away and behind the other (i.e., the ground) if its texture is accreted or deleted at the boundary between the regions, irrespective of region and boundary velocities. Consequently, a region with moving texture appears farther away than a stationary region if the boundary is stationary, but it appears closer (i.e., the figure) if the boundary is moving coherently with the moving texture. A computational model of visual areas V1 and V2 shows how interactions between orientation- and direction-selective cells first create a motion-defined boundary and then signal kinetic occlusion at that boundary. Activation of model occlusion detectors tuned to a particular velocity results in the model assigning the adjacent surface with a matching velocity to the far depth. A weak speed-depth bias brings faster-moving texture regions forward in depth in the absence of occlusion (shearing motion). These processes together reproduce human psychophysical reports of depth ordering for key cases of kinetic occlusion displays.

Dynamic coding of border-ownership in visual cortex.

Humans are capable of rapidly determining whether regions in a visual scene appear as figures in the foreground or as background, yet how figure-ground segregation occurs in the primate visual system is unknown. Figures in the environment are perceived to own their borders, and recent neurophysiology has demonstrated that certain cells in primate visual area V2 have border-ownership selectivity. We present a dynamic model based on physiological data that indicates areas V1, V2, and V4 act as an interareal network to determine border-ownership. Our model predicts that competition between curvature- sensitive cells in V4 that have on-surround receptive fields of different sizes can determine likely figure locations and rapidly propagate the information interareally to V2 border-ownership cells that receive contrast information from V1. In the model border-ownership is an emergent property produced by the dynamic interactions between V1, V2, and V4, one which could not be determined by any single cortical area alone.

Active gaze control improves optic flow-based segmentation and steering.

An observer traversing an environment actively relocates gaze to fixate objects. Evidence suggests that gaze is frequently directed toward the center of an object considered as target but more likely toward the edges of an object that appears as an obstacle. We suggest that this difference in gaze might be motivated by specific patterns of optic flow that are generated by either fixating the center or edge of an object. To support our suggestion we derive an analytical model that shows: Tangentially fixating the outer surface of an obstacle leads to strong flow discontinuities that can be used for flow-based segmentation. Fixation of the target center while gaze and heading are locked without head-, body-, or eye-rotations gives rise to a symmetric expansion flow with its center at the point being approached, which facilitates steering toward a target. We conclude that gaze control incorporates ecological constraints to improve the robustness of steering and collision avoidance by actively generating flows appropriate to solve the task.

Modeling the influence of optic flow on grid cell firing in the absence of other cues1.

Information from the vestibular, sensorimotor, or visual systems can affect the firing of grid cells recorded in entorhinal cortex of rats. Optic flow provides information about the rat's linear and rotational velocity and, thus, could influence the firing pattern of grid cells. To investigate this possible link, we model parts of the rat's visual system and analyze their capability in estimating linear and rotational velocity. In our model a rat is simulated to move along trajectories recorded from rat's foraging on a circular ground platform. Thus, we preserve the intrinsic statistics of real rats' movements. Visual image motion is analytically computed for a spherical camera model and superimposed with noise in order to model the optic flow that would be available to the rat. This optic flow is fed into a template model to estimate the rat's linear and rotational velocities, which in turn are fed into an oscillatory interference model of grid cell firing. Grid scores are reported while altering the flow noise, tilt angle of the optical axis with respect to the ground, the number of flow templates, and the frequency used in the oscillatory interference model. Activity patterns are compatible with those of grid cells, suggesting that optic flow can contribute to their firing.

Representation of motion onset and offset in an augmented Barlow-Levick model of motion detection.

Kinetic occlusion produces discontinuities in the optic flow field, whose perception requires the detection of an unexpected onset or offset of otherwise predictably moving or stationary contrast patches. Many cells in primate visual cortex are directionally selective for moving contrasts, and recent reports suggest that this selectivity arises through the inhibition of contrast signals moving in the cells' null direction, as in the rabbit retina. This nulling inhibition circuit (Barlow-Levick) is here extended to also detect motion onsets and offsets. The selectivity of extended circuit units, measured as a peak evidence accumulation response to motion onset/offset compared to the peak response to constant motion, is analyzed as a function of stimulus speed. Model onset cells are quiet during constant motion, but model offset cells activate during constant motion at slow speeds. Consequently, model offset cell speed tuning is biased towards higher speeds than onset cell tuning, similarly to the speed tuning of cells in the middle temporal area when exposed to speed ramps. Given a population of neurons with different preferred speeds, this asymmetry addresses a behavioral paradox-why human subjects in a simple reaction time task respond more slowly to motion offsets than onsets for low speeds, even though monkey neuron firing rates react more quickly to the offset of a preferred stimulus than to its onset.

Neural dynamics of object-based multifocal visual spatial attention and priming: object cueing, useful-field-of-view, and crowding.

How are spatial and object attention coordinated to achieve rapid object learning and recognition during eye movement search? How do prefrontal priming and parietal spatial mechanisms interact to determine the reaction time costs of intra-object attention shifts, inter-object attention shifts, and shifts between visible objects and covertly cued locations? What factors underlie individual differences in the timing and frequency of such attentional shifts? How do transient and sustained spatial attentional mechanisms work and interact? How can volition, mediated via the basal ganglia, influence the span of spatial attention? A neural model is developed of how spatial attention in the where cortical stream coordinates view-invariant object category learning in the what cortical stream under free viewing conditions. The model simulates psychological data about the dynamics of covert attention priming and switching requiring multifocal attention without eye movements. The model predicts how "attentional shrouds" are formed when surface representations in cortical area V4 resonate with spatial attention in posterior parietal cortex (PPC) and prefrontal cortex (PFC), while shrouds compete among themselves for dominance. Winning shrouds support invariant object category learning, and active surface-shroud resonances support conscious surface perception and recognition. Attentive competition between multiple objects and cues simulates reaction-time data from the two-object cueing paradigm. The relative strength of sustained surface-driven and fast-transient motion-driven spatial attention controls individual differences in reaction time for invalid cues. Competition between surface-driven attentional shrouds controls individual differences in detection rate of peripheral targets in useful-field-of-view tasks. The model proposes how the strength of competition can be mediated, though learning or momentary changes in volition, by the basal ganglia. A new explanation of crowding shows how the cortical magnification factor, among other variables, can cause multiple object surfaces to share a single surface-shroud resonance, thereby preventing recognition of the individual objects.

The animat: new frontiers in whole brain modeling.

The researchers at Boston University (BU)'s Neuromorphics Laboratory, part of the National Science Foundation (NSF)-sponsored Center of Excellence for Learning in Education, Science, and Technology (CELEST), are working in collaboration with the engineers and scientists at Hewlett-Packard (HP) to implement neural models of intelligent processes for the next generation of dense, low-power, computer hardware that will use memristive technology to bring data closer to the processor where computation occurs. The HP and BU teams are jointly designing an optimal infrastructure, simulation, and software platform to build an artificial brain. The resulting Cog Ex Machina (Cog) software platform has been successfully used to implement a large-scale, multicomponent brain system that is able to simulate some key rat behavioral results in a virtual environment and has been applied to control robotic platforms as they learn to interact with their environment.

A motion pooling model of visually guided navigation explains human behavior in the presence of independently moving objects.

Humans accurately judge their direction of heading when translating in a rigid environment, unless independently moving objects (IMOs) cross the observer's focus of expansion (FoE). Studies show that an IMO on a laterally moving path that maintains a fixed distance with respect to the observer (non-approaching; C. S. Royden & E. C. Hildreth, 1996) biases human heading estimates differently from an IMO on a lateral path that gets closer to the observer (approaching; W. H. Warren & J. A. Saunders, 1995). C. S. Royden (2002) argued that differential motion operators in primate brain area MT explained both data sets, concluding that differential motion was critical to human heading estimation. However, neurophysiological studies show that motion pooling cells, but not differential motion cells, in MT project to heading-sensitive cells in MST (V. K. Berezovskii & R. T. Born, 2000). It is difficult to reconcile differential motion heading models with these neurophysiological data. We generate motion sequences that mimic those viewed by human subjects. Model MT pools over V1; units in model MST perform distance-weighted template matching and compete in a recurrent heading representation layer. Our model produces heading biases of the same direction and magnitude as humans through a peak shift in model MSTd without using differential motion operators, maintaining consistency with known primate neurophysiology.

A model of motion transparency processing with local center-surround interactions and feedback.

Motion transparency occurs when multiple coherent motions are perceived in one spatial location. Imagine, for instance, looking out of the window of a bus on a bright day, where the world outside the window is passing by and movements of passengers inside the bus are reflected in the window. The overlay of both motions at the window leads to motion transparency, which is challenging to process. Noisy and ambiguous motion signals can be reduced using a competition mechanism for all encoded motions in one spatial location. Such a competition, however, leads to the suppression of multiple peak responses that encode different motions, as only the strongest response tends to survive. As a solution, we suggest a local center-surround competition for population-encoded motion directions and speeds. Similar motions are supported, and dissimilar ones are separated, by representing them as multiple activations, which occurs in the case of motion transparency. Psychophysical findings, such as motion attraction and repulsion for motion transparency displays, can be explained by this local competition. Besides this local competition mechanism, we show that feedback signals improve the processing of motion transparency. A discrimination task for transparent versus opaque motion is simulated, where motion transparency is generated by superimposing large field motion patterns of either varying size or varying coherence of motion. The model's perceptual thresholds with and without feedback are calculated. We demonstrate that initially weak peak responses can be enhanced and stabilized through modulatory feedback signals from higher stages of processing.

A new motion illusion based on competition between two kinds of motion processing units: the accordion grating.

Parametric psychophysical investigations are reported for two related illusory effects that occur when viewing an elementary square-wave grating while making "back and forth" head movements along the projection line. Observers report a non-rigid distortion of the pattern, including: (i) an expansion in a direction perpendicular to the stripes, and (ii) a perceived curvature of the stripes. We investigated these two phenomena independently. The first depends on the classical physiological aperture problem that confronts early cells in the vision system. Interactions between ambiguous and unambiguous motion signals, generated at line interiors and line ends, respectively, can explain why the perceived expansion occurs only in directions perpendicular to the stripes. A simple model is presented and successfully tested by a nulling psychophysical experiment with four subjects. The experiment varies key stimulus attributes that generate ambiguous and unambiguous motion signals. Regarding the illusory curvature, a differential geometry model of the optics of our display, which identifies a non-classical three-dimensional (3D) aperture problem, is proposed (Yazdanbakhsh & Gori, 2011). We tested that model by implementing its closed form prediction of distortion to design displays for a second psychophysical experiment that also uses a nulling technique. Results from four subjects allow the quantification of the degree of perceived curvature as a function of speed, distance and stimulus type (blurred vs. unblurred grating) and are compatible with the predictions of the model.

Perceiving opponent hues in color induction displays.

According to Hering's color theory, certain hues (red vs green and blue vs yellow) are mutually exclusive as components of a single color; consequently a color cannot be perceived as reddish-green or bluish-yellow. The goal of our study is to test this key postulate of the opponent color theory. Using the method of adjustment, our observers determine the boundaries of chromatic zones in a red-green continuum. We demonstrate on two distinct stimulus sets, one formed using a chromatic grid and neon spreading and the other based on solid colored regions, that the chromatic contrast of a purple surround over a red figure results in perception of 'forbidden' reddish-green colors. The observed phenomenon can be understood as resulting from the construction of a virtual filter, a process that bypasses photoreceptor summation and permits forbidden color combinations. Showing that opponent hue combinations, previously reported only under artificial image stabilization, can be present in normal viewing conditions offers new approaches for the experimental study of the dimensionality and structure of perceptual color space.

The effect of contrast intensity and polarity in the achromatic watercolor effect.

The watercolor effect (WCE) is a filling-in phenomenon in a region demarcated by two thin abutting lines. The perceived chromaticity of the region is similar to that of the interior line. We develop a series of achromatic WCE stimuli to induce lightness changes analogous to the induced chromaticity in the chromatic version of the WCE. We use a variation of the paired-comparison paradigm to quantify the induced lightness of the filled-in regions to regions with real luminance variations. The luminance of the inner line is fixed, while the luminance of the outer line varies across stimuli. Data from seven subjects (five naive) confirm that an achromatic WCE exists. Moreover, outer lines with both high and low luminances can generate a WCE with an inner line of a moderate luminance. All subjects show a single peak of the effect strength for both polarity conditions, which is never at the extreme luminance levels. Most subjects show an inverted U curve for effect strength as a function of the contrast of the outer lines against the background. Results suggest that the contrast difference between the outer line and the inner line affects the existence and the strength of the achromatic WCE in a nonlinear way.